Project - Computer Vision - 1

by HARI SAMYNAATH S

DOMAIN: Botanical research
• CONTEXT: University X is currently undergoing some research involving understanding the characteristics of plant and plant seedlings at various stages of growth. They already have have invested on curating sample images. They require an automation which can create a classifier capable of determining a plant's species from a photo
• DATA DESCRIPTION: The dataset comprises of images from 12 plant species. Source: https://www.kaggle.com/c/plant-seedlings-classification/data
• PROJECT OBJECTIVE: University’s management require an automation which can create a classi ier capable of determining a plant's species from a photo

Steps and tasks:

1. Import the data. Analyse the dimensions of the data. Visualise the data.

from PIL import Image
import os
from joblib import Parallel, delayed
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.layers import Dense, Dropout, BatchNormalization, Flatten, Concatenate, InputLayer
from tensorflow.keras.utils import plot_model
from tensorflow.keras import optimizers, regularizers
from tensorboard.plugins.hparams import api as hp
from tensorflow.keras.models import load_model
from tensorflow.keras import backend
import tensorflow as tf

from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Ridge, Lasso
from sklearn.svm import SVR

import numpy as np
import pandas as pd
import datetime, os, pickle
import matplotlib.pyplot as plt
tf.test.gpu_device_name()

2022-04-10 22:40:52.539744: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 22:40:52.539936: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 22:40:52.540091: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 22:40:52.540254: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 22:40:52.540373: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 22:40:52.540462: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /device:GPU:0 with 3044 MB memory:  -> device: 0, name: Quadro P1000, pci bus id: 0000:01:00.0, compute capability: 6.1

'/device:GPU:0'

lets review the image dimensions in the train data

%%time
def sizes(impath):
    im = Image.open(impath)
    return(im.size)

def inFolders(fldpath):
    imsizes = Parallel(n_jobs=-1,pre_dispatch='1.5*n_jobs',verbose=0
                        )(delayed(sizes
                                 )(fldpath+"/"+imgname
                                  ) for imgname in os.listdir(fldpath))
    return imsizes

dims = Parallel(n_jobs=2,pre_dispatch='1.5*n_jobs',verbose=0
               )(delayed(inFolders
                        )("./train/"+folder) for folder in os.listdir("./train"))

CPU times: user 27.4 ms, sys: 64.5 ms, total: 91.9 ms
Wall time: 626 ms

dimdata = []
for species in dims:
    dimdata.extend(species)
print("total number of images in training data: %d"%len(dimdata))

total number of images in training data: 4767

pxls = [tf.multiply(h,w).numpy() for h,w in dimdata ]
h = [h for h,w in dimdata]
w = [w for h,w in dimdata]
print("dims of largest image: ",dimdata[tf.argmax(pxls)])
print("dims of widest image: ",dimdata[tf.argmax(w)])
print("dims of tallest image: ",dimdata[tf.argmax(h)])
print("dims of smallest image: ",dimdata[tf.argmin(pxls)])
print("dims of leanest image: ",dimdata[tf.argmin(w)])
print("dims of shortest image: ",dimdata[tf.argmin(h)])

dims of largest image:  (3991, 3457)
dims of widest image:  (3991, 3457)
dims of tallest image:  (3991, 3457)
dims of smallest image:  (49, 49)
dims of leanest image:  (49, 49)
dims of shortest image:  (49, 49)

2022-04-10 13:50:56.225377: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 13:50:56.225686: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 13:50:56.225896: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 13:50:56.226513: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 13:50:56.226748: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 13:50:56.226946: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 13:50:56.227230: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 13:50:56.227448: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-10 13:50:56.227613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3044 MB memory:  -> device: 0, name: Quadro P1000, pci bus id: 0000:01:00.0, compute capability: 6.1

Lets visualise 5 images randomly from each

for cat in os.listdir("./train"):
    plt.figure(figsize=(20,50))
    for i,img in enumerate(np.random.choice(os.listdir('./train/'+cat),5,replace=False)):
        img = Image.open('./train/'+cat+'/'+img).resize((200,200))
        plt.subplot(1,5,i+1)
        plt.imshow(img)
        plt.title(cat)
    plt.show()

Steps and tasks:

1. Design, train tune and test the best AIML image classifier model using:
   • Supervised learning algorithms
   • Neural networks
   • CNN
   

# irrespective of the model to be deployed, the models need consistent input dimensions
# hence set up data preprocessing mechanism
# the least dimensions found in the dataset was 49x49
# using imagedatagenerator to produce augmented images of dimensions 49x49
datagen = ImageDataGenerator(rescale = 1./255, rotation_range=20,
                                   shear_range = 0.2, zoom_range = 0.2,
                                   width_shift_range = 0.1, height_shift_range = 0.1,
                                   horizontal_flip = True, vertical_flip = True,
                                   data_format = 'channels_last', validation_split = 0.1)

trainset = datagen.flow_from_directory('./train', target_size = (49,49), 
                                                  batch_size=32, subset='training',class_mode = 'categorical')

validationset = datagen.flow_from_directory('./train', target_size = (49,49), 
                                                  batch_size=32, subset='validation',class_mode = 'categorical')

# resize prediction image to similar size
test_datagen = ImageDataGenerator(rescale = 1./255)
img = Image.open('Predict.png')
testset = test_datagen.flow(tf.expand_dims(img.resize((49,49)),axis=0))

Found 4294 images belonging to 12 classes.
Found 473 images belonging to 12 classes.

#Clear any logs from previous runs try: !rm -rf ./logs/ os.mkdir('logs') except: os.mkdir('logs') fpath = "/media/harinaathan/Barracuda/HARI SAMYNAATH S/Anaconda_workspace/GLAIML_course/09 - ComputerVision/3 - week Project/logs/" file = open(fpath+"CNN_tuning.har",'wb') # reset log file file.close() #above file creation & log reset to be commented after setting up a tuning framework

fpath = "/media/harinaathan/Barracuda/HARI SAMYNAATH S/Anaconda_workspace/GLAIML_course/09 - ComputerVision/3 - week Project/logs/"

Transfer Learning from Resnet50

Trusting in the capabilities of Resnet50, lets perform a transfer learning from the Resnet50 CNN
later lets build customized Neural Networks & a Machine learning algorithm, in an attempt to acheive the accuracy of CNN based transfer learning technique.

lets functionally connect the resnet output to a series of Dense & BatchNormalisation layers before connecting to a softmax layer via a Dropout
also lets take a parallel stream of a slightly different heirarchy and concat both the outputs before the softmax
to choose the best combination of architecture, lets tryout few epochs
we would also combine the hyperparameter search on the go

# create parameter options
nNodes = [1024,512,256,128,64]
nBranches = hp.HParam('Branches',hp.Discrete([1,2,3,4]))
dropOut = hp.HParam('DropOutRatio',hp.Discrete([0.2,0.4,0.6,0.8]))
lr = hp.HParam('LearningRate',hp.Discrete([1e-2,1e-3,1e-4]))
skip = hp.HParam('skipConnection',hp.Discrete([True,False]))
dRate = hp.HParam('DecayRate',hp.Discrete([0.1,0.3,0.5]))

# define the hyper parameter of the architecture
branchNodes = []
for branch in nBranches.domain.values: #HParam Branches
    nodeList = []
    for i in range(branch):
        nodeList.extend([nNodes[i]]) #HParam nodes
    branchNodes.append(nodeList)

# create a grid of hyperparameters
pgrid = [[nNode,dropout,lrt,skp,dr]
         for nNode in branchNodes
         for dropout in dropOut.domain.values
         for lrt in lr.domain.values
         for skp in skip.domain.values
         for dr in dRate.domain.values
        ]

print("number of grid points: ",len(pgrid))

number of grid points:  288

gridRuns = [] # reset local log

# read the tuning log (if any)
file = open(fpath+"CNN_tuning.har",'rb')
read = True
while read:
    try:
        gridRuns.append(pickle.load(file))
    except:
        read = False
file.close()

if len(gridRuns)>0:
    grids = [i[:-3] for i in gridRuns]
else:
    grids = []

# delete pre-evaluated gridpoints (if any)
for gp in grids: 
    try:
        pgrid.remove(gp)
    except:
        pass

print("grid points remaining: ",len(pgrid))

grid points remaining:  255

halt here for_ this error do not_ re-run the cell below the hypertuning is complete for 300 search instances

# customised learning rate scheduler
# considering slope of validation loss
class lossGradientLR(tf.keras.callbacks.Callback):
    def __init__(self,initial_learning_rate,patience,factor,verbose=False):
        super(lossGradientLR, self).__init__()
        self.initial_learning_rate = initial_learning_rate
        self.patience = patience
        self.factor = factor
        self.verbose=verbose
        self.loss=[]
        self.lr=[initial_learning_rate]
        self.lastAdjust = 0
        
    def Gradient_schedule(self,epoch):
        flag1 = False
        flag1 = (epoch-self.lastAdjust)>self.patience # min epoch before adjust
        
        flag2 = False
        try:
            flag2 = (self.loss[-1]-self.loss[-2])>0 and (self.loss[-2]-self.loss[-3])>0
            # if two continuous epochs have increasing validation loss
        except:
            pass
        
        # slope of validation loss
        lossGrad = (self.loss[-1]-self.loss[self.lastAdjust])
        lossGrad/= len(self.loss[self.lastAdjust:])
        
        # if slope of validation loss not descending at -0.25 slope or flag2
        if (flag1 and lossGrad>-0.25) or (flag2):
            self.lr.append(max(self.lr[-1]*self.factor,1e-7)) #min lr 1e-7
            self.lastAdjust = epoch # update last adjust epoch
        
        if self.verbose:
            print("\nslopeOfLoss:%.2f"%lossGrad," @ epoch:%d"%epoch, "---- learning_rate:",self.lr[-1])
        
    def on_epoch_end(self,epoch,logs={}):
        """on end of each epoch"""
        # fetch validation loss
        self.loss.append(logs.get('val_loss'))
        #invoke learning rate calculator
        self.Gradient_schedule(epoch)
        
    def on_epoch_begin(self,epoch,logs={}):
        """beginning of each epoch"""
        # update model learning rate
        tf.keras.backend.set_value(self.model.optimizer.lr, self.lr[-1])

# report learning rate to tensorboard
# due to custom lr_scheduler
class LRTensorBoard(tf.keras.callbacks.TensorBoard):
    def __init__(self, log_dir, **kwargs):
        super().__init__(log_dir=log_dir, **kwargs)

    def on_epoch_end(self, epoch, logs=None):
        logs = logs or {}
        logs.update({'learning_rate': backend.eval(self.model.optimizer.learning_rate)})
        super().on_epoch_end(epoch, logs)

%%time
# perform random grid search
epochs = 20
nSearch = 5

for niter,gp in enumerate(np.random.randint(0,len(pgrid),nSearch)):
    
    print("grid :",len(gridRuns)+1)
    
    # Random consistency seed
    np.random.seed(129)
    tf.random.set_seed(129)
    
    # attempting transfer learning from ResNet50
    resnet = ResNet50(include_top=False, weights='imagenet', input_shape=(49,49,3), pooling='max')
    
    _nBranch = len(pgrid[gp][0])
    _nodes = pgrid[gp][0] # HParam nNodes
    
    # create model architecture
    X = []
    # First: Flatten layer deriving inputs from resnet
    flat = Flatten()(resnet.output)
    # Second: parallel branches
    for i in range(_nBranch): # HParam Branches
        branch = Dense(_nodes[i],activation='relu')(flat)
        branch = BatchNormalization()(branch)
        X.extend([branch])
    # Optional skip connection bypass to branches
    if pgrid[gp][3]:
        X.extend([flat]) # HParam SkipConnection
    # Third: Concat all above branches and skip connection
    x = Concatenate()(X)
    # Fourth : dense 36 nodes
    x = Dense(36, activation="relu")(x)
    # Fifth : Dropout HParam
    x = Dropout(pgrid[gp][1])(x) # HParam Dropout 
    # Final & Sixth : Softmax classification result
    x = Dense(12, activation='softmax')(x)
    
    model = Model(resnet.input,x)
    model.trainable=True
    resnetlayers=len(resnet.layers)
    for i,layer in enumerate(model.layers):
        if i<resnetlayers:
            layer.trainable = False # freeze resnet layers
        else:
            layer.trainable=True # unfreeze custom layers
    
    mname = "htuneCNN_%03d_"%(len(gridRuns)+1) + datetime.datetime.now().strftime("%d%b_%I%M%P")
    
    # create tensorboard logs & callbacks
    logdir = os.path.join("logs", mname )
    tensorboard = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
    
    # initiate learning rate reporter
    lReport = LRTensorBoard(logdir)

    verbose = max(0,1-niter) # chat once, then go silent
    
    # initiate learning rate callback
    lr_schedule = lossGradientLR(pgrid[gp][2],# HParam learning rate
                                 patience=5,verbose=bool(verbose),
                                 factor=pgrid[gp][4])# HParam decayRate
    
    # define optimsers and compile the model
    optim = optimizers.RMSprop(learning_rate=pgrid[gp][2],# HParam learning rate
                               rho=0.9, momentum=0.1, epsilon=1e-07)
    model.compile(optimizer = optim, loss = 'categorical_crossentropy', metrics = ['accuracy'])
    
    # summarise the HParams
    hparams = dict(nBranches=len(pgrid[gp][0]),
                   skip=pgrid[gp][3],
                   dropOut=pgrid[gp][1],
                   lr=pgrid[gp][2],
                   dRate=pgrid[gp][4]
                  )
    
    # callback for HParam writting
    hypertune = hp.KerasCallback(logdir,hparams)
    
    # model fitting
    with tf.device('/device:GPU:0'):
        logger = model.fit(trainset, epochs = epochs,
                           verbose = verbose, steps_per_epoch = 10, # limited learning for hypertuning
                           validation_data = validationset, validation_steps = 5,
                           callbacks=[tensorboard,lr_schedule,lReport,hypertune])
    
    # obtain peak results
    p_acc,p_epoch = np.max(logger.history.get('val_accuracy')),np.argmax(logger.history.get('val_accuracy'))
    
    # summarise hparams with results
    with tf.summary.create_file_writer('logs/hparam_tuning').as_default():
        hp.hparams(hparams)
        accuracy = p_acc
        tf.summary.scalar('accuracy',accuracy,step=1)
    
    # log the results
    storage = []
    storage.extend(pgrid[gp])
    storage.extend([p_acc,p_epoch])
    storage.extend([mname])
    
    file = open(fpath+"CNN_tuning.har",'ab')
    pickle.dump(storage,file)
    file.close()
    
    gridRuns = [] # reset local log

    # read the logs
    file = open(fpath+"CNN_tuning.har",'rb')
    read = True
    while read:
        try:
            gridRuns.append(pickle.load(file))
        except:
            read = False
    file.close()

grid : 31
Epoch 1/20
10/10 [==============================] - ETA: 0s - loss: 2.9930 - accuracy: 0.1094
slopeOfLoss:0.00  @ epoch:0 ---- learning_rate: 0.001
10/10 [==============================] - 13s 1s/step - loss: 2.9930 - accuracy: 0.1094 - val_loss: 2.7477 - val_accuracy: 0.1250 - learning_rate: 0.0010
Epoch 2/20
10/10 [==============================] - ETA: 0s - loss: 2.7742 - accuracy: 0.1469
slopeOfLoss:-0.07  @ epoch:1 ---- learning_rate: 0.001
10/10 [==============================] - 9s 933ms/step - loss: 2.7742 - accuracy: 0.1469 - val_loss: 2.6022 - val_accuracy: 0.1187 - learning_rate: 0.0010
Epoch 3/20
10/10 [==============================] - ETA: 0s - loss: 2.6437 - accuracy: 0.0969
slopeOfLoss:-0.09  @ epoch:2 ---- learning_rate: 0.001
10/10 [==============================] - 8s 812ms/step - loss: 2.6437 - accuracy: 0.0969 - val_loss: 2.4743 - val_accuracy: 0.1375 - learning_rate: 0.0010
Epoch 4/20
10/10 [==============================] - ETA: 0s - loss: 2.5371 - accuracy: 0.1187
slopeOfLoss:-0.04  @ epoch:3 ---- learning_rate: 0.001
10/10 [==============================] - 8s 872ms/step - loss: 2.5371 - accuracy: 0.1187 - val_loss: 2.5958 - val_accuracy: 0.0875 - learning_rate: 0.0010
Epoch 5/20
10/10 [==============================] - ETA: 0s - loss: 2.5463 - accuracy: 0.1469
slopeOfLoss:-0.06  @ epoch:4 ---- learning_rate: 0.001
10/10 [==============================] - 9s 944ms/step - loss: 2.5463 - accuracy: 0.1469 - val_loss: 2.4623 - val_accuracy: 0.0500 - learning_rate: 0.0010
Epoch 6/20
10/10 [==============================] - ETA: 0s - loss: 2.6051 - accuracy: 0.1063
slopeOfLoss:-0.04  @ epoch:5 ---- learning_rate: 0.001
10/10 [==============================] - 8s 875ms/step - loss: 2.6051 - accuracy: 0.1063 - val_loss: 2.4861 - val_accuracy: 0.0938 - learning_rate: 0.0010
Epoch 7/20
10/10 [==============================] - ETA: 0s - loss: 2.4629 - accuracy: 0.1844
slopeOfLoss:-0.03  @ epoch:6 ---- learning_rate: 0.0003
10/10 [==============================] - 8s 823ms/step - loss: 2.4629 - accuracy: 0.1844 - val_loss: 2.5331 - val_accuracy: 0.1375 - learning_rate: 0.0010
Epoch 8/20
10/10 [==============================] - ETA: 0s - loss: 2.4489 - accuracy: 0.1469
slopeOfLoss:0.08  @ epoch:7 ---- learning_rate: 8.999999999999999e-05
10/10 [==============================] - 8s 882ms/step - loss: 2.4489 - accuracy: 0.1469 - val_loss: 2.7028 - val_accuracy: 0.1375 - learning_rate: 3.0000e-04
Epoch 9/20
10/10 [==============================] - ETA: 0s - loss: 2.3465 - accuracy: 0.1781
slopeOfLoss:0.00  @ epoch:8 ---- learning_rate: 2.6999999999999996e-05
10/10 [==============================] - 8s 813ms/step - loss: 2.3465 - accuracy: 0.1781 - val_loss: 2.7117 - val_accuracy: 0.1125 - learning_rate: 9.0000e-05
Epoch 10/20
10/10 [==============================] - ETA: 0s - loss: 2.3638 - accuracy: 0.1844
slopeOfLoss:-0.04  @ epoch:9 ---- learning_rate: 2.6999999999999996e-05
10/10 [==============================] - 8s 834ms/step - loss: 2.3638 - accuracy: 0.1844 - val_loss: 2.6317 - val_accuracy: 0.1312 - learning_rate: 2.7000e-05
Epoch 11/20
10/10 [==============================] - ETA: 0s - loss: 2.3526 - accuracy: 0.1562
slopeOfLoss:-0.01  @ epoch:10 ---- learning_rate: 2.6999999999999996e-05
10/10 [==============================] - 7s 805ms/step - loss: 2.3526 - accuracy: 0.1562 - val_loss: 2.6737 - val_accuracy: 0.1000 - learning_rate: 2.7000e-05
Epoch 12/20
10/10 [==============================] - ETA: 0s - loss: 2.3826 - accuracy: 0.1625
slopeOfLoss:-0.05  @ epoch:11 ---- learning_rate: 2.6999999999999996e-05
10/10 [==============================] - 8s 816ms/step - loss: 2.3826 - accuracy: 0.1625 - val_loss: 2.5034 - val_accuracy: 0.1562 - learning_rate: 2.7000e-05
Epoch 13/20
10/10 [==============================] - ETA: 0s - loss: 2.3715 - accuracy: 0.1781
slopeOfLoss:-0.02  @ epoch:12 ---- learning_rate: 2.6999999999999996e-05
10/10 [==============================] - 8s 830ms/step - loss: 2.3715 - accuracy: 0.1781 - val_loss: 2.6059 - val_accuracy: 0.1187 - learning_rate: 2.7000e-05
Epoch 14/20
10/10 [==============================] - ETA: 0s - loss: 2.3623 - accuracy: 0.1844
slopeOfLoss:-0.03  @ epoch:13 ---- learning_rate: 2.6999999999999996e-05
10/10 [==============================] - 8s 879ms/step - loss: 2.3623 - accuracy: 0.1844 - val_loss: 2.5202 - val_accuracy: 0.1250 - learning_rate: 2.7000e-05
Epoch 15/20
10/10 [==============================] - ETA: 0s - loss: 2.3601 - accuracy: 0.1875
slopeOfLoss:-0.03  @ epoch:14 ---- learning_rate: 8.099999999999999e-06
10/10 [==============================] - 8s 865ms/step - loss: 2.3601 - accuracy: 0.1875 - val_loss: 2.4794 - val_accuracy: 0.1562 - learning_rate: 2.7000e-05
Epoch 16/20
10/10 [==============================] - ETA: 0s - loss: 2.3749 - accuracy: 0.1656
slopeOfLoss:0.01  @ epoch:15 ---- learning_rate: 8.099999999999999e-06
10/10 [==============================] - 8s 804ms/step - loss: 2.3749 - accuracy: 0.1656 - val_loss: 2.5027 - val_accuracy: 0.1437 - learning_rate: 8.1000e-06
Epoch 17/20
10/10 [==============================] - ETA: 0s - loss: 2.3582 - accuracy: 0.1813
slopeOfLoss:0.02  @ epoch:16 ---- learning_rate: 2.4299999999999996e-06
10/10 [==============================] - 8s 903ms/step - loss: 2.3582 - accuracy: 0.1813 - val_loss: 2.5365 - val_accuracy: 0.1125 - learning_rate: 8.1000e-06
Epoch 18/20
10/10 [==============================] - ETA: 0s - loss: 2.3982 - accuracy: 0.1594
slopeOfLoss:-0.03  @ epoch:17 ---- learning_rate: 2.4299999999999996e-06
10/10 [==============================] - 8s 842ms/step - loss: 2.3982 - accuracy: 0.1594 - val_loss: 2.4788 - val_accuracy: 0.1562 - learning_rate: 2.4300e-06
Epoch 19/20
10/10 [==============================] - ETA: 0s - loss: 2.3682 - accuracy: 0.1406
slopeOfLoss:-0.02  @ epoch:18 ---- learning_rate: 2.4299999999999996e-06
10/10 [==============================] - 8s 873ms/step - loss: 2.3682 - accuracy: 0.1406 - val_loss: 2.4874 - val_accuracy: 0.1437 - learning_rate: 2.4300e-06
Epoch 20/20
10/10 [==============================] - ETA: 0s - loss: 2.3225 - accuracy: 0.2075
slopeOfLoss:-0.01  @ epoch:19 ---- learning_rate: 2.4299999999999996e-06
10/10 [==============================] - 8s 833ms/step - loss: 2.3225 - accuracy: 0.2075 - val_loss: 2.4785 - val_accuracy: 0.1187 - learning_rate: 2.4300e-06
grid : 32
grid : 33
grid : 34
grid : 35
CPU times: user 28min 47s, sys: 5min 18s, total: 34min 5s
Wall time: 14min 4s

# display the log in a convenient pandas frame
cols = ["nNodes","DropoutRatio","LearningRate","SkipConnection","DecayRate","PeakAccuracy","PeakEpoch","ModelName"]
display(pd.DataFrame(gridRuns,columns=cols).sort_values(by="PeakAccuracy",ascending=False).head())

	nNodes	DropoutRatio	LearningRate	SkipConnection	DecayRate	PeakAccuracy	PeakEpoch	ModelName
17	[1024, 512, 256, 128]	0.8	0.001	True	0.3	0.18750	13	htuneCNN_018_10Apr_0253pm
10	[1024, 512, 256, 128]	0.2	0.001	True	0.5	0.18750	16	htuneCNN_011_10Apr_0235pm
33	[1024, 512, 256]	0.6	0.001	True	0.1	0.18125	18	htuneCNN_034_10Apr_0348pm
4	[1024, 512, 256]	0.8	0.001	True	0.1	0.18125	10	htuneCNN_005_10Apr_0218pm
14	[1024, 512, 256]	0.8	0.010	True	0.3	0.17500	13	htuneCNN_015_10Apr_0245pm

# review & selection of the best model from tensorflow
plt.figure(figsize=(12,18));
plt.imshow(plt.imread('htuneCNN_log.png'));
plt.figure(figsize=(12,18));
plt.imshow(plt.imread('htuneCNN_bestmodel_1.png'));
plt.figure(figsize=(12,18));
plt.imshow(plt.imread('htuneCNN_bestmodel_2.png'));
plt.figure(figsize=(12,18));
plt.imshow(plt.imread('htuneCNN_bestmodel_3.png'));
plt.figure(figsize=(12,18));
plt.imshow(plt.imread('htuneCNN_bestmodel_4.png'));

# chosen top grids
grids[4],grids[9],grids[18],grids[33]

([[1024, 512, 256], 0.8, 0.001, True, 0.1],
 [[1024, 512, 256, 128], 0.2, 0.0001, True, 0.1],
 [[1024, 512, 256], 0.2, 0.01, True, 0.3],
 [[1024, 512, 256], 0.6, 0.001, True, 0.1])

%%time
# lets put the 4 top model architectures to full dataset
refine = [grids[4],grids[9],grids[18],grids[33]]

epochs = 50

for niter,gp in enumerate(refine):
    
    # Random consistency seed
    np.random.seed(129)
    tf.random.set_seed(129)
    
    # attempting transfer learning from ResNet50
    resnet = ResNet50(include_top=False, weights='imagenet', input_shape=(49,49,3), pooling='max')
    
    _nBranch = len(gp[0])
    _nodes = gp[0] # HParam nNodes
    
    # create model architecture
    X = []
    # First: Flatten layer deriving inputs from resnet
    flat = Flatten()(resnet.output)
    # Second: parallel branches
    for i in range(_nBranch): # HParam Branches
        branch = Dense(_nodes[i],activation='relu')(flat)
        branch = BatchNormalization()(branch)
        X.extend([branch])
    # Optional skip connection bypass to branches
    if gp[3]:
        X.extend([flat]) # HParam SkipConnection
    # Third: Concat all above branches and skip connection
    x = Concatenate()(X)
    # Fourth : dense 36 nodes
    x = Dense(36, activation="relu")(x)
    # Fifth : Dropout HParam
    x = Dropout(gp[1])(x) # HParam Dropout 
    # Final & Sixth : Softmax classification result
    x = Dense(12, activation='softmax')(x)
    
    model = Model(resnet.input,x)
    model.trainable=True
    resnetlayers=len(resnet.layers)
    for i,layer in enumerate(model.layers):
        if i<resnetlayers:
            layer.trainable = False # freeze resnet layers
        else:
            layer.trainable=True # unfreeze custom layers
    
    mname = "refineCNN_%03d_"%(niter+1) + datetime.datetime.now().strftime("%d%b_%I%M%P")
    
    # create tensorboard logs & callbacks
    logdir = os.path.join("logs", mname )
    tensorboard = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
    
    # initiate learning rate reporter
    lReport = LRTensorBoard(logdir)

    verbose = 0 #silent
    
    # save best model weights
    saveFile = 'logs/'+mname+'_weights.{epoch:02d}-{val_accuracy:.2f}.h5'
    saveBest = tf.keras.callbacks.ModelCheckpoint(filepath=saveFile,monitor='val_accuracy',
                                              mode='max',save_best_only=True)
    
    # early stop callback
    earlyStop = tf.keras.callbacks.EarlyStopping(monitor='val_accuracy',
                                                 min_delta=0.01,patience=15,restore_best_weights=True)
    
    # initiate learning rate callback
    lr_schedule = lossGradientLR(gp[2],# HParam learning rate
                                 patience=5,verbose=bool(verbose),
                                 factor=gp[4])# HParam decayRate
    
    # define optimsers and compile the model
    optim = optimizers.RMSprop(learning_rate=gp[2],# HParam learning rate
                               rho=0.9, momentum=0.1, epsilon=1e-07)
    model.compile(optimizer = optim, loss = 'categorical_crossentropy', metrics = ['accuracy'])
    
    # model fitting
    with tf.device('/device:GPU:0'):
        logger = model.fit(trainset, epochs = epochs, verbose = verbose, 
                           validation_data = validationset,
                           callbacks=[tensorboard,lr_schedule,lReport,saveBest,earlyStop])
    
    # obtain peak results
    p_acc,p_epoch = np.max(logger.history.get('val_accuracy')),np.argmax(logger.history.get('val_accuracy'))
    
    # restore the best weights
    savefiles=[]
    # search the saveFiles
    for root, dirs, files in os.walk('./logs'):
        for file in files:
            if file.startswith(mname+'_weights') and file.endswith('.h5'):
                savefiles.append(file)
    # choose the best epoch (latest saveBest)
    bestweights = sorted(savefiles,reverse=True)[0]
    # load the weights
    model.load_weights('./logs/'+bestweights)
    
    # save the model
    model.save('./'+mname+'.h5')
    model.save_weights('./'+mname+'_weights.h5')
    
    # log the results
    storage = []
    storage.extend(gp)
    storage.extend([p_acc,p_epoch])
    storage.extend([mname])
    
    file = open(fpath+"CNN_tuning.har",'ab')
    pickle.dump(storage,file)
    file.close()
    
    gridRuns = [] # reset local log

    # read the logs
    file = open(fpath+"CNN_tuning.har",'rb')
    read = True
    while read:
        try:
            gridRuns.append(pickle.load(file))
        except:
            read = False
    file.close()

2022-04-10 17:05:23.446733: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.38GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

CPU times: user 1h 16min 28s, sys: 5min 23s, total: 1h 21min 52s
Wall time: 1h 3min 46s

# review for the 4 model performances
plt.figure(figsize=(15,45));
plt.subplot(131);
plt.imshow(plt.imread('refine_accuracy.png'));
plt.subplot(132);
plt.imshow(plt.imread('refine_loss.png'));
plt.subplot(133);
plt.imshow(plt.imread('refine_lr.png'));

only refineCNN_002 with grid[9] sustained the iterations
lets try to unfreeze the resnet layers and relearn the model

# prepare for further training
# Random consistency seed
np.random.seed(129)
tf.random.set_seed(129)

# warm start / transfer learning
refineCNN = load_model('./refineCNN_002_10Apr_0516pm.h5')
refineCNN.load_weights('./refineCNN_002_10Apr_0516pm_weights.h5')

refineCNN.trainable=True
firstTrainLayer=None
for i,layer in enumerate(model.layers):
    if layer.name.startswith('conv5'): # unfreeze 5th convolutional group
        if firstTrainLayer == None:
            firstTrainLayer=i
    try:
        if i>=firstTrainLayer:
            layer.trainable=True
    except:
        layer.trainable=False

for i,layer in enumerate(model.layers):
    print("%03d"%i," %-25s"%layer.name,"---",layer.trainable)

000  input_45                  --- False
001  conv1_pad                 --- False
002  conv1_conv                --- False
003  conv1_bn                  --- False
004  conv1_relu                --- False
005  pool1_pad                 --- False
006  pool1_pool                --- False
007  conv2_block1_1_conv       --- False
008  conv2_block1_1_bn         --- False
009  conv2_block1_1_relu       --- False
010  conv2_block1_2_conv       --- False
011  conv2_block1_2_bn         --- False
012  conv2_block1_2_relu       --- False
013  conv2_block1_0_conv       --- False
014  conv2_block1_3_conv       --- False
015  conv2_block1_0_bn         --- False
016  conv2_block1_3_bn         --- False
017  conv2_block1_add          --- False
018  conv2_block1_out          --- False
019  conv2_block2_1_conv       --- False
020  conv2_block2_1_bn         --- False
021  conv2_block2_1_relu       --- False
022  conv2_block2_2_conv       --- False
023  conv2_block2_2_bn         --- False
024  conv2_block2_2_relu       --- False
025  conv2_block2_3_conv       --- False
026  conv2_block2_3_bn         --- False
027  conv2_block2_add          --- False
028  conv2_block2_out          --- False
029  conv2_block3_1_conv       --- False
030  conv2_block3_1_bn         --- False
031  conv2_block3_1_relu       --- False
032  conv2_block3_2_conv       --- False
033  conv2_block3_2_bn         --- False
034  conv2_block3_2_relu       --- False
035  conv2_block3_3_conv       --- False
036  conv2_block3_3_bn         --- False
037  conv2_block3_add          --- False
038  conv2_block3_out          --- False
039  conv3_block1_1_conv       --- False
040  conv3_block1_1_bn         --- False
041  conv3_block1_1_relu       --- False
042  conv3_block1_2_conv       --- False
043  conv3_block1_2_bn         --- False
044  conv3_block1_2_relu       --- False
045  conv3_block1_0_conv       --- False
046  conv3_block1_3_conv       --- False
047  conv3_block1_0_bn         --- False
048  conv3_block1_3_bn         --- False
049  conv3_block1_add          --- False
050  conv3_block1_out          --- False
051  conv3_block2_1_conv       --- False
052  conv3_block2_1_bn         --- False
053  conv3_block2_1_relu       --- False
054  conv3_block2_2_conv       --- False
055  conv3_block2_2_bn         --- False
056  conv3_block2_2_relu       --- False
057  conv3_block2_3_conv       --- False
058  conv3_block2_3_bn         --- False
059  conv3_block2_add          --- False
060  conv3_block2_out          --- False
061  conv3_block3_1_conv       --- False
062  conv3_block3_1_bn         --- False
063  conv3_block3_1_relu       --- False
064  conv3_block3_2_conv       --- False
065  conv3_block3_2_bn         --- False
066  conv3_block3_2_relu       --- False
067  conv3_block3_3_conv       --- False
068  conv3_block3_3_bn         --- False
069  conv3_block3_add          --- False
070  conv3_block3_out          --- False
071  conv3_block4_1_conv       --- False
072  conv3_block4_1_bn         --- False
073  conv3_block4_1_relu       --- False
074  conv3_block4_2_conv       --- False
075  conv3_block4_2_bn         --- False
076  conv3_block4_2_relu       --- False
077  conv3_block4_3_conv       --- False
078  conv3_block4_3_bn         --- False
079  conv3_block4_add          --- False
080  conv3_block4_out          --- False
081  conv4_block1_1_conv       --- False
082  conv4_block1_1_bn         --- False
083  conv4_block1_1_relu       --- False
084  conv4_block1_2_conv       --- False
085  conv4_block1_2_bn         --- False
086  conv4_block1_2_relu       --- False
087  conv4_block1_0_conv       --- False
088  conv4_block1_3_conv       --- False
089  conv4_block1_0_bn         --- False
090  conv4_block1_3_bn         --- False
091  conv4_block1_add          --- False
092  conv4_block1_out          --- False
093  conv4_block2_1_conv       --- False
094  conv4_block2_1_bn         --- False
095  conv4_block2_1_relu       --- False
096  conv4_block2_2_conv       --- False
097  conv4_block2_2_bn         --- False
098  conv4_block2_2_relu       --- False
099  conv4_block2_3_conv       --- False
100  conv4_block2_3_bn         --- False
101  conv4_block2_add          --- False
102  conv4_block2_out          --- False
103  conv4_block3_1_conv       --- False
104  conv4_block3_1_bn         --- False
105  conv4_block3_1_relu       --- False
106  conv4_block3_2_conv       --- False
107  conv4_block3_2_bn         --- False
108  conv4_block3_2_relu       --- False
109  conv4_block3_3_conv       --- False
110  conv4_block3_3_bn         --- False
111  conv4_block3_add          --- False
112  conv4_block3_out          --- False
113  conv4_block4_1_conv       --- False
114  conv4_block4_1_bn         --- False
115  conv4_block4_1_relu       --- False
116  conv4_block4_2_conv       --- False
117  conv4_block4_2_bn         --- False
118  conv4_block4_2_relu       --- False
119  conv4_block4_3_conv       --- False
120  conv4_block4_3_bn         --- False
121  conv4_block4_add          --- False
122  conv4_block4_out          --- False
123  conv4_block5_1_conv       --- False
124  conv4_block5_1_bn         --- False
125  conv4_block5_1_relu       --- False
126  conv4_block5_2_conv       --- False
127  conv4_block5_2_bn         --- False
128  conv4_block5_2_relu       --- False
129  conv4_block5_3_conv       --- False
130  conv4_block5_3_bn         --- False
131  conv4_block5_add          --- False
132  conv4_block5_out          --- False
133  conv4_block6_1_conv       --- False
134  conv4_block6_1_bn         --- False
135  conv4_block6_1_relu       --- False
136  conv4_block6_2_conv       --- False
137  conv4_block6_2_bn         --- False
138  conv4_block6_2_relu       --- False
139  conv4_block6_3_conv       --- False
140  conv4_block6_3_bn         --- False
141  conv4_block6_add          --- False
142  conv4_block6_out          --- False
143  conv5_block1_1_conv       --- True
144  conv5_block1_1_bn         --- True
145  conv5_block1_1_relu       --- True
146  conv5_block1_2_conv       --- True
147  conv5_block1_2_bn         --- True
148  conv5_block1_2_relu       --- True
149  conv5_block1_0_conv       --- True
150  conv5_block1_3_conv       --- True
151  conv5_block1_0_bn         --- True
152  conv5_block1_3_bn         --- True
153  conv5_block1_add          --- True
154  conv5_block1_out          --- True
155  conv5_block2_1_conv       --- True
156  conv5_block2_1_bn         --- True
157  conv5_block2_1_relu       --- True
158  conv5_block2_2_conv       --- True
159  conv5_block2_2_bn         --- True
160  conv5_block2_2_relu       --- True
161  conv5_block2_3_conv       --- True
162  conv5_block2_3_bn         --- True
163  conv5_block2_add          --- True
164  conv5_block2_out          --- True
165  conv5_block3_1_conv       --- True
166  conv5_block3_1_bn         --- True
167  conv5_block3_1_relu       --- True
168  conv5_block3_2_conv       --- True
169  conv5_block3_2_bn         --- True
170  conv5_block3_2_relu       --- True
171  conv5_block3_3_conv       --- True
172  conv5_block3_3_bn         --- True
173  conv5_block3_add          --- True
174  conv5_block3_out          --- True
175  max_pool                  --- True
176  flatten_44                --- True
177  dense_239                 --- True
178  dense_240                 --- True
179  dense_241                 --- True
180  batch_normalization_151   --- True
181  batch_normalization_152   --- True
182  batch_normalization_153   --- True
183  concatenate_44            --- True
184  dense_242                 --- True
185  dropout_44                --- True
186  dense_243                 --- True

epochs = 100
gp = grids[9]

mname = "unfreezeCNN_001_" + datetime.datetime.now().strftime("%d%b_%I%M%P")

# create tensorboard logs & callbacks
logdir = os.path.join("logs", mname )
tensorboard = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)

# initiate learning rate reporter
lReport = LRTensorBoard(logdir)

verbose = 0 #silent

# save best model weights
saveFile = 'logs/'+mname+'_weights.{epoch:02d}-{val_accuracy:.2f}.h5'
saveBest = tf.keras.callbacks.ModelCheckpoint(filepath=saveFile,monitor='val_accuracy',
                                              mode='max',save_best_only=True)

# early stop callback
earlyStop = tf.keras.callbacks.EarlyStopping(monitor='val_accuracy',
                                             min_delta=0.01,patience=15,restore_best_weights=True)

# initiate learning rate callback
lr_schedule = lossGradientLR(gp[2],# HParam learning rate
                             patience=5,verbose=bool(verbose),
                             factor=gp[4])# HParam decayRate

# define optimsers and compile the model
optim = optimizers.RMSprop(learning_rate=gp[2],# HParam learning rate
                           rho=0.9, momentum=0.1, epsilon=1e-07)
refineCNN.compile(optimizer = optim, loss = 'categorical_crossentropy', metrics = ['accuracy'])

# model fitting
with tf.device('/device:GPU:0'):
    logger = refineCNN.fit(trainset, epochs = epochs, verbose = verbose, 
                       validation_data = validationset,
                       callbacks=[tensorboard,lr_schedule,lReport,saveBest,earlyStop])

# obtain peak results
p_acc,p_epoch = np.max(logger.history.get('val_accuracy')),np.argmax(logger.history.get('val_accuracy'))

# restore the best weights
savefiles=[]
# search the saveFiles
for root, dirs, files in os.walk('./logs'):
    for file in files:
        if file.startswith(mname+'_weights') and file.endswith('.h5'):
            savefiles.append(file)
# choose the best epoch (latest saveBest)
bestweights = sorted(savefiles,reverse=True)[0]
# load the weights
refineCNN.load_weights('./logs/'+bestweights)

# save the model
refineCNN.save('./unfreezeCNN_001_.h5')
refineCNN.save_weights('./unfreezeCNN_001_weights.h5')

# log the results
storage = []
storage.extend(gp)
storage.extend([p_acc,p_epoch])
storage.extend([mname])

file = open(fpath+"CNN_tuning.har",'ab')
pickle.dump(storage,file)
file.close()

gridRuns = [] # reset local log

# read the logs
file = open(fpath+"CNN_tuning.har",'rb')
read = True
while read:
    try:
        gridRuns.append(pickle.load(file))
    except:
        read = False
file.close()

2022-04-10 18:55:46.145314: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2022-04-10 18:55:46.167149: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.41GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2022-04-10 18:55:46.335716: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.30GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2022-04-10 18:55:46.357768: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2022-04-10 18:56:01.748054: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2022-04-10 18:56:01.758978: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.30GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2022-04-10 18:56:01.876294: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.28GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2022-04-10 18:56:01.890948: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

# review for the Conv5 unFreezed model
plt.figure(figsize=(15,45));
plt.subplot(131);
plt.imshow(plt.imread('unFreeze_accuracy.png'));
plt.subplot(132);
plt.imshow(plt.imread('unFreeze_loss.png'));
plt.subplot(133);
plt.imshow(plt.imread('unFreeze_lr.png'));

the unfreezeCNN_001 has achieved remarkable 0.92 accuracy
lets put it to prediction of the given image

unFreezeCNN = load_model('./unfreezeCNN_001_.h5')
unFreezeCNN.load_weights('./unfreezeCNN_001_weights.h5')
# predict for the predict.png loaded via ImageDataGenerator as testset
prediction = unFreezeCNN.predict(testset)
# process the softmax output
classIndex = np.argmax(prediction>0.5)
seedling = list(trainset.class_indices.keys())[classIndex]
# display the result and image
plt.figure();
plt.title(seedling);
plt.imshow(testset.next()[0]);
plt.figure();
plt.title('prediction probabilities from softmax');
plt.barh(list(trainset.class_indices.keys()),prediction[0],log=True);

2022-04-10 19:42:31.156728: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

the Resnet50 based CNN have predicted the test image correctly

Artificial Neural Networks model

lets build a few neural network modelling functions that will build, learn, evaluate

# lets create parameter gridpoints
hidden_layers = np.arange(2,6,1)
hidden_neurons = np.arange(50,500,50)
learning_rates = [1e-3,1e-4,1e-5]
Lambdas = [0.25,0.5,0.75]
dos = np.arange(0.2,0.7,0.1)

pgrid = [[hl,hn,lr,lm,do] 
           for hl in hidden_layers
           for hn in hidden_neurons
           for lr in learning_rates
           for lm in Lambdas
           for do in dos]

print("number of grid points: ",len(pgrid))

number of grid points:  1620

file = open(fpath+"NN_tuning.har",'wb') # reset log file file.close() # above file not re-created after setting up a good hypertuning framework # hence above line commented

gridRuns = [] # reset local log

# read the tuning log (if any)
file = open(fpath+"NN_tuning.har",'rb')
read = True
while read:
    try:
        gridRuns.append(pickle.load(file))
    except:
        read = False
file.close()

if len(gridRuns)>0:
    grids = [i[:-3] for i in gridRuns]
else:
    grids = []

# delete pre-evaluated gridpoints (if any)
for gp in grids: 
    try:
        pgrid.remove(gp)
    except:
        pass

print("grid points remaining: ",len(pgrid))

grid points remaining:  1557

%%time
# perform random grid search
epochs = 20
nSearch = 5
verbose = 0

for niter,gp in enumerate(np.random.randint(0,len(pgrid),nSearch)):
    
    print("grid :",len(gridRuns)+1)
    
    [hl,hln,lr,lmbda,do]=pgrid[gp]
    
    # Random consistency seed
    np.random.seed(129)
    tf.random.set_seed(129)
    
    # Build a Sequential model
    NN = Sequential()
    NN.add(InputLayer(input_shape=(49,49,3)))
    NN.add(Flatten())
    init = tf.keras.initializers.GlorotNormal(seed=129)
    NN.add(Dense(hln,activation='relu',
               kernel_initializer=init,
               kernel_regularizer=regularizers.l2(lmbda)))
    NN.add(BatchNormalization())
    for i in range(hl-1):
        NN.add(Dense(hln,activation='relu',
                        kernel_initializer=init,
                        kernel_regularizer=regularizers.l2(lmbda)))
        NN.add(BatchNormalization())
    NN.add(Dropout(do))
    NN.add(Dense(12, activation='softmax'))
    
    mname = "htuneNN_%03d_"%(len(gridRuns)+1) + datetime.datetime.now().strftime("%d%b_%I%M%P")
    
    # create tensorboard logs & callbacks
    logdir = os.path.join("logs", mname )
    tensorboard = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
    
    # initiate learning rate reporter
    lReport = LRTensorBoard(logdir)
    
    # initiate learning rate callback
    lr_schedule = lossGradientLR(lr,patience=5,verbose=bool(verbose),factor=0.25)
    
    # define optimsers and compile the model
    optim = optimizers.RMSprop(learning_rate=lr, rho=0.9, momentum=0.1, epsilon=1e-07)
    NN.compile(optimizer = optim, loss = 'categorical_crossentropy', metrics = ['accuracy'])
    
    # summarise the HParams
    hparams = dict(hiddenLayers=hl,
                   neuronsPerLayer=hln,
                   learningRate=lr,
                   Lambda=lmbda,
                   DropOut=do
                  )
    
    # callback for HParam writting
    hypertune = hp.KerasCallback(logdir,hparams)
    
    # model fitting
    with tf.device('/device:GPU:0'):
        logger = NN.fit(trainset, epochs = epochs,
                           verbose = 0, steps_per_epoch = 10, # limited learning for hypertuning
                           validation_data = validationset, validation_steps = 5,
                           callbacks=[tensorboard,hypertune,lr_schedule,lReport])
    
    # obtain peak results
    p_acc,p_epoch = np.max(logger.history.get('val_accuracy')),np.argmax(logger.history.get('val_accuracy'))
    
    # summarise hparams with results
    with tf.summary.create_file_writer('logs/hparam_tuning').as_default():
        hp.hparams(hparams)
        accuracy = p_acc
        tf.summary.scalar('accuracy',accuracy,step=1)
    
    # log the results
    storage = []
    storage.extend(pgrid[gp])
    storage.extend([p_acc,p_epoch])
    storage.extend([mname])
    
    file = open(fpath+"NN_tuning.har",'ab')
    pickle.dump(storage,file)
    file.close()
    
    gridRuns = [] # reset local log

    # read the logs
    file = open(fpath+"NN_tuning.har",'rb')
    read = True
    while read:
        try:
            gridRuns.append(pickle.load(file))
        except:
            read = False
    file.close()
    
if len(gridRuns)>0:
    grids = [i[:-3] for i in gridRuns]
else:
    grids = []

# display the log in a convenient pandas frame
cols = ["HiddenLayers","NeuronsPerLayer","LearningRate","Lambda","DropOut","PeakAccuracy","PeakEpoch","ModelName"]
display(pd.DataFrame(gridRuns,columns=cols).sort_values(by="PeakAccuracy",ascending=False).head())

grid : 72
grid : 73
grid : 74
grid : 75
grid : 76

	HiddenLayers	NeuronsPerLayer	LearningRate	Lambda	DropOut	PeakAccuracy	PeakEpoch	ModelName
65	5	150	0.0001	0.25	0.3	0.528541	95	trialNN_066_09Apr_0953pm
70	2	400	0.0001	0.25	0.6	0.287500	7	htuneNN_071_10Apr_0755pm
64	5	150	0.0001	0.25	0.3	0.270613	39	trialNN_065_09Apr_0940pm
21	3	200	0.0001	0.75	0.4	0.225000	16	htuneNN_022_09Apr_0806pm
29	4	100	0.0010	0.25	0.3	0.218750	10	htuneNN_030_09Apr_0821pm

CPU times: user 9min 19s, sys: 25.6 s, total: 9min 44s
Wall time: 8min 7s

# eliminate long run models from results
htuneNNresults = pd.DataFrame(gridRuns,columns=cols).sort_values(by="PeakAccuracy",ascending=False)
htuneNNresults.loc[htuneNNresults["PeakEpoch"]<=20].head()

	HiddenLayers	NeuronsPerLayer	LearningRate	Lambda	DropOut	PeakAccuracy	PeakEpoch	ModelName
70	2	400	0.0001	0.25	0.6	0.28750	7	htuneNN_071_10Apr_0755pm
21	3	200	0.0001	0.75	0.4	0.22500	16	htuneNN_022_09Apr_0806pm
29	4	100	0.0010	0.25	0.3	0.21875	10	htuneNN_030_09Apr_0821pm
7	4	100	0.0010	0.25	0.5	0.21875	12	htuneNN_008_09Apr_0737pm
35	2	350	0.0010	0.50	0.2	0.21875	1	htuneNN_036_09Apr_0831pm

%%time
# trying out 
# more steps, more steps_per_epoch, full validation steps and learning rate decay
# with one of the best learner

epochs = 100
grid=grids[70]
[hl,hln,lr,lmbda,do]=grid

# Random consistency seed
np.random.seed(129)
tf.random.set_seed(129)

# Build a Sequential model
NN = Sequential()
NN.add(InputLayer(input_shape=(49,49,3)))
NN.add(Flatten())
init = tf.keras.initializers.GlorotNormal(seed=129)
NN.add(Dense(hln,activation='relu',
             kernel_initializer=init,
             kernel_regularizer=regularizers.l2(lmbda)))
NN.add(BatchNormalization())
for i in range(hl-1):
    NN.add(Dense(hln,activation='relu',
                 kernel_initializer=init,
                 kernel_regularizer=regularizers.l2(lmbda)))
    NN.add(BatchNormalization())
NN.add(Dropout(do))
NN.add(Dense(12, activation='softmax'))

mname = "refineNN_001_" + datetime.datetime.now().strftime("%d%b_%I%M%P")

# create tensorboard logs & callbacks
logdir = os.path.join("logs", mname )
tensorboard = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)

# initiate learning rate reporter
lReport = LRTensorBoard(logdir)
    
# initiate learning rate callback
lr_schedule = lossGradientLR(lr,patience=5,verbose=bool(verbose),factor=0.2)

# define optimsers and compile the model
optim = optimizers.RMSprop(learning_rate=lr, rho=0.9, momentum=0.1, epsilon=1e-07)
NN.compile(optimizer = optim, loss = 'categorical_crossentropy', metrics = ['accuracy'])

# early stop callback
earlyStop = tf.keras.callbacks.EarlyStopping(monitor='val_accuracy',
                                             min_delta=0.01,patience=15,restore_best_weights=True)

# save best model weights
saveFile = 'logs/'+mname+'_weights.{epoch:02d}-{val_accuracy:.2f}.h5'
saveBest = tf.keras.callbacks.ModelCheckpoint(filepath=saveFile,monitor='val_accuracy',
                                              mode='max',save_best_only=True)

# model fitting
with tf.device('/device:GPU:0'):
    logger = NN.fit(trainset, epochs = epochs,verbose = 0,
                    validation_data = validationset,
                    callbacks=[tensorboard,lr_schedule,lReport,earlyStop,saveBest])
    
# obtain peak results
p_acc,p_epoch = np.max(logger.history.get('val_accuracy')),np.argmax(logger.history.get('val_accuracy'))

# restore the best weights
savefiles=[]
# search the saveFiles
for root, dirs, files in os.walk('./logs'):
    for file in files:
        if file.startswith(mname+'_weights') and file.endswith('.h5'):
            savefiles.append(file)
# choose the best epoch (latest saveBest)
bestweights = sorted(savefiles,reverse=True)[0]
# load the weights
NN.load_weights('./logs/'+bestweights)

# save the model
NN.save('./refineNN_001.h5')
NN.save_weights('./refineNN_001_weights.h5')
    
# log the results
storage = []
storage.extend(grid)
storage.extend([p_acc,p_epoch])
storage.extend([mname])

file = open(fpath+"NN_tuning.har",'ab')
pickle.dump(storage,file)
file.close()

gridRuns = [] # reset local log

# read the logs
file = open(fpath+"NN_tuning.har",'rb')
read = True
while read:
    try:
        gridRuns.append(pickle.load(file))
    except:
        read = False
file.close()

if len(gridRuns)>0:
    grids = [i[:-3] for i in gridRuns]
else:
    grids = []

CPU times: user 50min 49s, sys: 1min, total: 51min 49s
Wall time: 49min 56s

# review for the refineNN_001 model
plt.figure(figsize=(15,45));
plt.subplot(131);
plt.imshow(plt.imread('refineNN_1_accuracy.png'));
plt.subplot(132);
plt.imshow(plt.imread('refineNN_1_loss.png'));
plt.subplot(133);
plt.imshow(plt.imread('refineNN_1_lr.png'));

refineNN = load_model('./refineNN_001.h5')
refineNN.load_weights('./refineNN_001_weights.h5')
# predict for the predict.png loaded via ImageDataGenerator as testset
prediction = refineNN.predict(testset)
# process the softmax output
classIndex = np.argmax(prediction)
seedling = list(trainset.class_indices.keys())[classIndex]
# display the result and image
plt.figure();
plt.title(seedling);
plt.imshow(testset.next()[0]);
plt.figure();
plt.title('prediction probabilities from softmax');
plt.barh(list(trainset.class_indices.keys()),prediction[0],log=True);

though the predicted class is correct,
even the softmax output is not so peaky at the desired output class
this is direct effect of the poor accuracy of trained model

ML model

lets attempt a basic ML model

# lets create a train, tune & validation set for a ML model
# Random consistency seed
np.random.seed(129)
tf.random.set_seed(129)

# tuning set
tuneY=[]
tuneX=[]
for i in range(5):
    batch = trainset.next()
    for y in batch[1]:
        tuneY.append(np.argmax(y))
    for img in batch[0]:
        im = []
        for row in img:
            for col in row:
                im.extend(col)
        tuneX.append(im)
        
# validation set
valY=[]
valX=[]
for i in range(3):
    batch = trainset.next()
    for y in batch[1]:
        valY.append(np.argmax(y))
    for img in batch[0]:
        im = []
        for row in img:
            for col in row:
                im.extend(col)
        valX.append(im)
        
# training set
trainY=[]
trainX=[]
for i in range(len(trainset)-5-3):
    batch = trainset.next()
    for y in batch[1]:
        trainY.append(np.argmax(y))
    for img in batch[0]:
        im = []
        for row in img:
            for col in row:
                im.extend(col)
        trainX.append(im)

# review len of the dataset
len(tuneY),len(valY),len(trainY)

(160, 96, 4038)

# features in each datapoint
len(tuneX[0])

7203

# prediction test image predict.png
predImg = testset.next()[0]
predictX = []
for row in predImg:
    for col in row:
        predictX.extend(col)
predictX = np.reshape(predictX,(1,7203))

# the images are already scaled between 0 to 1
# and the images do not have an null / nan values
# hence no preprocessing is necessary
# except for the fact that this dataset posses CurseOfDimensionality as there only 4038 datapoints for 7203 features
# yet on experimental basis lets try a ML model

# since there are 12 classes to be predicted, and since we have orgainsed our Y as a vector ranging from 0 to 11
# we will be using regression models to predict the class

%%time
# base model
lm = LinearRegression(n_jobs=-1)
lm.fit(trainX,trainY)

CPU times: user 55.6 s, sys: 972 ms, total: 56.6 s
Wall time: 15.6 s

LinearRegression(n_jobs=-1)

print("training score: ",lm.score(trainX,trainY))
print("validation score: ",lm.score(valX, valY))

training score:  0.9999967691971685
validation score:  -5.503412813408733

thats extreme overfitting model

# predict based on linear regression
prediction = max(0,min(11,round(lm.predict(predictX)[0]))) # bounding the prediction to (0,11)
seedling = list(trainset.class_indices.keys())[prediction]
print("predicted seedling :",seedling)

predicted seedling : Sugar beet

as expected from the accuracy scores, it has predicted incorrectly

# lets try regularised Ridge model
rm = Ridge(alpha=800)
rm.fit(trainX,trainY)
print("training score: ",rm.score(trainX,trainY))
print("validation score: ",rm.score(valX, valY))
prediction = max(0,min(11,round(rm.predict(predictX)[0])))
seedling = list(trainset.class_indices.keys())[prediction]
print("predicted seedling :",seedling)

training score:  0.12376589275317795
validation score:  0.027234687264746982
predicted seedling : Loose Silky-bent

# reference of classes
trainset.class_indices

{'Black-grass': 0,
 'Charlock': 1,
 'Cleavers': 2,
 'Common Chickweed': 3,
 'Common wheat': 4,
 'Fat Hen': 5,
 'Loose Silky-bent': 6,
 'Maize': 7,
 'Scentless Mayweed': 8,
 'Shepherds Purse': 9,
 'Small-flowered Cranesbill': 10,
 'Sugar beet': 11}

that's pretty close to the desired result
but the result is mere fluke as the accuracy scores suggests

# lets try Lasso regression
lsm = Lasso(alpha = 5)
lsm.fit(trainX,trainY)
print("training score: ",lsm.score(trainX,trainY))
print("validation score: ",lsm.score(valX, valY))
prediction = max(0,min(11,round(lsm.predict(predictX)[0])))
seedling = list(trainset.class_indices.keys())[prediction]
print("predicted seedling :",seedling)

training score:  0.0
validation score:  -0.0028711428938446026
predicted seedling : Loose Silky-bent

not much of a good result

%%time
# try baseline SVR
svrm = SVR()
svrm.fit(trainX,trainY)
print("training score: ",svrm.score(trainX,trainY))
print("validation score: ",svrm.score(valX, valY))
prediction = max(0,min(11,round(svrm.predict(predictX)[0])))
seedling = list(trainset.class_indices.keys())[prediction]
print("predicted seedling :",seedling)

training score:  0.27432568768148524
validation score:  0.07086922873978319
predicted seedling : Loose Silky-bent
CPU times: user 13min 21s, sys: 1.74 s, total: 13min 23s
Wall time: 3min 24s

very much similar results

Steps and tasks:

1. Compare the results from the above step along with your detailed observations on the best performing algorithm and why/how it outperforms other algorithms in this case.
   
2. Pickle the best performing model.
   
3. Import the the test image [ from the “ Prediction” folder ] to predict the class. Display the image. Use the best trained image classi ier model to predict the class.
   

1. comparison and discussion to be found below
2. the best models were pickled in respective sections
3. the prediction shall be refered in the respective sections and teh best prediction will be repeated below

# compare all results
prediction = unFreezeCNN.predict(testset)
# CNN model
classIndex = np.argmax(prediction>0.5)
seedling = list(trainset.class_indices.keys())[classIndex]
plt.figure();
plt.title('CNN model');
plt.barh(list(trainset.class_indices.keys()),prediction[0],log=True);

prediction = refineNN.predict(testset)
# NN Model
classIndex = np.argmax(prediction)
seedling = list(trainset.class_indices.keys())[classIndex]
plt.figure();
plt.title('NN model');
plt.barh(list(trainset.class_indices.keys()),prediction[0],log=True);

print("linear regression validation accuracy",lm.score(valX, valY))
print("Lasso model validation accuracy",rm.score(valX, valY))
print("Ridge model validation accuracy",lsm.score(valX, valY))
print("support vector regression validation accuracy",svrm.score(valX, valY))

linear regression validation accuracy -5.503412813408733
Lasso model validation accuracy 0.027234687264746982
Ridge model validation accuracy -0.0028711428938446026
support vector regression validation accuracy 0.07086922873978319

it is evident that the CNN model outperforms all models
this is advantage of the following key points
1) imagenet learned weights - transfer learning
2) convolutions has more sense of spatial corelation
3) this resnet50 is 175 layer deep by itself

# predict using CNN
prediction = unFreezeCNN.predict(testset)
# process the softmax output
classIndex = np.argmax(prediction>0.5)
seedling = list(trainset.class_indices.keys())[classIndex]
# display the result and image
plt.figure();
plt.title(seedling);
plt.imshow(testset.next()[0]);